{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 项目名称：理解媒体、探究未来之路\n",
    "- 数据加值宣言：本项目产出按媒体前沿趋势研究和对内容创作的研究，挖掘公众号“腾讯媒体研究院”的117页数据，以解决人们对媒体行业的深究，理解当下媒体的发展趋势，为未来就业方向打下基础。\n",
    "\n",
    "# 数据最小可用产品\n",
    "- 读者可以根据当下媒体热词（如短视频、直播带货、爆款等），寻找到相关文章进行阅读，以便更好进行内容创作！\n",
    "\n",
    "# 挖掘微信公众号的基本信息\n",
    "- 挖掘到704条数据，包含文章标题、发布时间、文章链接、分类的信息。\n",
    "- 读者可以根据标题、分类查找自己感兴趣的文章，进行借鉴和学习。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备工作\n",
    "- 选择对微信公众号“腾讯媒体研究院”进行数据抓取"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 从中选择公众号“腾讯媒体研究院”进行数据抓取\n",
    "'''\n",
    "阿里研究院\n",
    "阿里健康\n",
    "阿里巴巴商学院\n",
    "阿里数据\n",
    "\n",
    "腾讯金融科技\n",
    "腾讯研究院\n",
    "腾讯媒体研究院\n",
    "腾讯云启研究院\n",
    "酷鹅用户研究院\n",
    "'''\n",
    "公众号 = \"腾讯媒体研究院\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "fn = { \"output\" : { \"公众号_htm_snippets\": \"data_raw_src/公众号_htm_snippets_{公众号}.tsv\",\n",
    "                    \"公众号_df\": \"data_raw_src/公众号_df_{公众号}.tsv\",\n",
    "                    \"公众号_数据_xlsx\": \"data_sets/公众号_url_{公众号}_数据.xlsx\"\n",
    "                  }\n",
    "      }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 采集公众号（selenium）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from lxml.html import fromstring\n",
    "import time\n",
    "from random import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用Selenium\n",
    "* 要更改 opts.binary_location 至自己本地的Chrome浏览器，建议portable\n",
    "* Chrome浏览器 和 chromedriver.exe要同版本号到小数后一位\n",
    "* 要确保可以 开启浏览器机器人\n",
    "* 要确保浏览器机器人 可以打开网页 driver.get(\"https://mp.weixin.qq.com\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\ipykernel_launcher.py:18: DeprecationWarning: use options instead of chrome_options\n"
     ]
    }
   ],
   "source": [
    "# 导入模块 下载chromedriver（与chrome同版本），放在chrome.exe同一目录\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.desired_capabilities import DesiredCapabilities\n",
    "\n",
    "\n",
    "opts = webdriver.ChromeOptions()\n",
    "opts.add_argument('--no-sandbox')#解决DevToolsActivePort文件不存在的报错\n",
    "opts.add_argument('window-size=1920x3000') #指定浏览器分辨率\n",
    "opts.add_argument('--disable-gpu') #谷歌文档提到需要加上一这个属性来规避bug\n",
    "opts.add_argument('--hide-scrollbars') #隐藏滚动条, 应对些特殊页面\n",
    "#opts.add_argument('blink-settings=imagesEnabled=false') #不加载图片, 提升速度\n",
    "#opts.add_argument('--headless') #浏览器不提供可视化页面. linux下如果系统不支持可视化不加这条会启动失败\n",
    "\n",
    "# 将chrome.exe的路径粘贴在此，并配置好环境变量\n",
    "opts.binary_location = r\"C:\\Users\\August\\AppData\\Local\\Google\\Chrome\\Application\\chrome.exe\" #\"H:\\_coding_\\Gitee\\InternetNewMedia\\CapstonePrj2016\\chromedriver.exe\"  \n",
    "\n",
    "# \"H:\\_coding_\\Gitee\\InternetNewMedia\\CapstonePrj2016\\chromedriver.exe\"  \n",
    "driver = webdriver.Chrome( chrome_options = opts) #desired_capabilities=caps,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.get(\"https://mp.weixin.qq.com\") # 自动打开微信公众平台"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 填表登入"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "selenium 的定位方法\n",
    "* find_element_by_id &ensp;&ensp;&ensp;  根据标签id定位\n",
    "* find_element_by_name   &ensp;&ensp;&ensp; 根据标签的name定位\n",
    "* find_element_by_xpath  &ensp;&ensp;&ensp; 根据xpath定位\n",
    "* find_element_by_link_text  &ensp;&ensp;&ensp; 通过文字链接来定位元素\n",
    "* find_element_by_partial_link_text  &ensp;&ensp;&ensp;  通过文字链接来定位元素\n",
    "* find_element_by_tag_name  &ensp;&ensp;&ensp;  根据标签的名字定位\n",
    "* find_element_by_class_name  &ensp;&ensp;&ensp; 通过class name 定位\n",
    "* find_element_by_css_selector  &ensp;&ensp;&ensp;  根据元素属性来定位"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 输入自己的账密\n",
    "payload =  {\"account\": \" \", \"password\": \" \"}\n",
    "driver.find_element_by_xpath('//div[@class=\"login__type__container login__type__container__scan\"]/a').click()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WebDriver 常用方法：\n",
    "* clear()清除文本\n",
    "* send_keys(values)模拟按键输入\n",
    "* click()模拟点击\n",
    "* submit模拟提交"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_element_by_xpath('//form[@class=\"login_form\"]//input[@name=\"account\"]').clear() # 清空账号\n",
    "driver.find_element_by_xpath('//form[@class=\"login_form\"]//input[@name=\"account\"]').send_keys(payload['account']) #模拟按键输入账号\n",
    "driver.find_element_by_xpath('//form[@class=\"login_form\"]//input[@name=\"password\"]').clear() # 清空密码\n",
    "driver.find_element_by_xpath('//form[@class=\"login_form\"]//input[@name=\"password\"]').send_keys(payload['password']) # 模拟按键输入密码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_element_by_xpath('//div[@class=\"login_btn_panel\"]/a').click() # 点击登录"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 点选单"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其他常用方法\n",
    "* size：返回元素的尺寸\n",
    "* text：获取元素的文本\n",
    "* get_attribute：获取属性值  &ensp;&ensp;&ensp; get_attribute('innerHTML')获取元素内的全部HTML\n",
    "* is_displayed()：设置该元素用户是否可见"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'展开'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 展开左边功能栏\n",
    "element = driver.find_element_by_xpath('//a[@id=\"m_open\"]')\n",
    "element.click()\n",
    "main_content = element.get_attribute('innerHTML')\n",
    "main_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.execute_script(\"window.scrollTo(0,document.body.scrollHeight)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://mp.weixin.qq.com/cgi-bin/appmsg?begin=0&count=10&t=media/appmsg_list&type=10&action=list&token=371019582&lang=zh_CN'"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 获取素材管理url\n",
    "element = driver.find_element_by_xpath('//li[@title[contains(.,\"素材管理\")]]/a') \n",
    "# main_content = element.get_attribute('innerHTML')\n",
    "# main_content\n",
    "url_素材管理= element.get_attribute(\"href\")\n",
    "url_素材管理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 模拟点击“素材管理”\n",
    "driver.get(url_素材管理)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 新建图文消息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 模拟点击“新建图文消息”\n",
    "element = driver.find_element_by_xpath('//*[text()[contains(.,\"新建图文消息\")]]') \n",
    "main_content = element.get_attribute('innerHTML')\n",
    "main_content\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['CDwindow-57FC9FF92B97067C3F50713406473383', 'CDwindow-81EF31A054FB551E6246F6AC465C0D06']\n"
     ]
    }
   ],
   "source": [
    "print (driver.window_handles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 新建图文消息开了另一分视窗，所以要切换 switch_to \n",
    "driver.switch_to.window(driver.window_handles[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 超链接"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                超链接              \n"
     ]
    }
   ],
   "source": [
    "# 模拟点击“超链接”\n",
    "element = driver.find_element_by_xpath('//*[text()[contains(.,\"超链接\")]]') \n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "选择其他公众号\n"
     ]
    }
   ],
   "source": [
    "# 点 选择其他公众号\n",
    "element = driver.find_element_by_xpath('//*[text()[contains(.,\"选择其他公众号\")]]') \n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 自动输入公众号名称——准备工作中的“腾讯媒体研究院”\n",
    "driver.find_element_by_xpath('//form//div[@class=\"inner_link_account_area\"]//input[@class=\"weui-desktop-form__input\"]').clear()\n",
    "driver.find_element_by_xpath('//form//div[@class=\"inner_link_account_area\"]//input[@class=\"weui-desktop-form__input\"]').send_keys(公众号)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<div class=\"weui-desktop-icon weui-desktop-icon__inputSearch weui-desktop-icon__small\"><!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <!----> <svg width=\"16\" height=\"16\" viewBox=\"0 0 16 16\" xmlns=\"http://www.w3.org/2000/svg\"><path d=\"M11.33 10.007l4.273 4.273a.502.502 0 0 1 .005.709l-.585.584a.499.499 0 0 1-.709-.004L10.046 11.3a6.278 6.278 0 1 1 1.284-1.294zm.012-3.729a5.063 5.063 0 1 0-10.127 0 5.063 5.063 0 0 0 10.127 0z\"></path></svg> <!----> <!----> <!----> <!----></div>\n"
     ]
    }
   ],
   "source": [
    "# 点放大镜搜\n",
    "element = driver.find_element_by_xpath('//button[@class=\"weui-desktop-icon-btn weui-desktop-search__btn\"]')\n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/cdSLicavGUr23RakfD1j8csT3LMhYpOQ21ECXeZMpic1k1dTBWprafgazS97J5JaVCHNQr4ib0cuSGVDWc9rJUiatQ/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">腾讯媒体研究院</strong> <i class=\"inner_link_account_wechat\">微信号：TencentMRI</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div></li><li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/gRl4QnJRXWbgpDlacGvtd3icLOwstUj5eIRYLZyPVUBGV1BZAGjve7t7IX5tvLibMic2mwkiam9IibyDv50M02RNjxQ/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">腾讯芒种特训营</strong> <i class=\"inner_link_account_wechat\">微信号：mz_camp</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">服务号</div></li><li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/HH8uRF2v4zyt0JfL70RQGMOGbagyprO18gIJaSJnvnicMvxquKlUMbRd7bhWicA36EGlxiaka0eVQjyiaKsAHwiaUdA/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">全媒派</strong> <i class=\"inner_link_account_wechat\">微信号：quanmeipai</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div></li><li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/twfPKR2ibmy2vqxPpFfqbahicn4aiaibpSicfHkWfpImatAsGicdOuVRzVyKeGI7cQl56Rp5yf7tuZAIJzd73tyOoHdA/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">腾讯多媒体实验室</strong> <i class=\"inner_link_account_wechat\">微信号：TencentAVLab</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">服务号</div></li><li class=\"inner_link_account_item\"><div class=\"weui-desktop-vm_primary\"><img src=\"http://mmsns.qpic.cn/mmsns/A0R6pEEjricaFJgpW6wLG3u0xRy0zzPPjrrfR5rhibPgyxEr3ibVhSODw/0\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">天津市多媒体教育技术研究会</strong> <i class=\"inner_link_account_wechat\">微信号：m-edu-m</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div></li>\n"
     ]
    }
   ],
   "source": [
    "# 搜索到关于“腾讯媒体研究院”的相关订阅号/服务号 列表\n",
    "element = driver.find_element_by_xpath('//ul[@class=\"inner_link_account_list\"]')\n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "公众号SERP = main_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 解析\n",
    "root = fromstring(公众号SERP)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "主 = root.xpath('//li[@class=\"inner_link_account_item\"]')\n",
    "\n",
    "account_list = []\n",
    "for e in 主:\n",
    "    account_nickname = e.xpath('./div/strong[@class=\"inner_link_account_nickname\"]')[0].text\n",
    "    account_wechat = e.xpath('./div/i[@class=\"inner_link_account_wechat\"]')[0].text\n",
    "    account_img = e.xpath('./div/img/@src')[0]\n",
    "    account = {\"nickname\": account_nickname, \"wechat\": account_wechat, \"img\": account_img,}\n",
    "    account_list.append(account)\n",
    "\n",
    "df_account = pd.DataFrame(account_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nickname</th>\n",
       "      <th>wechat</th>\n",
       "      <th>img</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>腾讯媒体研究院</td>\n",
       "      <td>微信号：TencentMRI</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/cdSLicavGUr23Ra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>腾讯芒种特训营</td>\n",
       "      <td>微信号：mz_camp</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/gRl4QnJRXWbgpDl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>全媒派</td>\n",
       "      <td>微信号：quanmeipai</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/HH8uRF2v4zyt0Jf...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>腾讯多媒体实验室</td>\n",
       "      <td>微信号：TencentAVLab</td>\n",
       "      <td>http://mmbiz.qpic.cn/mmbiz_png/twfPKR2ibmy2vqx...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>天津市多媒体教育技术研究会</td>\n",
       "      <td>微信号：m-edu-m</td>\n",
       "      <td>http://mmsns.qpic.cn/mmsns/A0R6pEEjricaFJgpW6w...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        nickname            wechat  \\\n",
       "0        腾讯媒体研究院    微信号：TencentMRI   \n",
       "1        腾讯芒种特训营       微信号：mz_camp   \n",
       "2            全媒派    微信号：quanmeipai   \n",
       "3       腾讯多媒体实验室  微信号：TencentAVLab   \n",
       "4  天津市多媒体教育技术研究会       微信号：m-edu-m   \n",
       "\n",
       "                                                 img  \n",
       "0  http://mmbiz.qpic.cn/mmbiz_png/cdSLicavGUr23Ra...  \n",
       "1  http://mmbiz.qpic.cn/mmbiz_png/gRl4QnJRXWbgpDl...  \n",
       "2  http://mmbiz.qpic.cn/mmbiz_png/HH8uRF2v4zyt0Jf...  \n",
       "3  http://mmbiz.qpic.cn/mmbiz_png/twfPKR2ibmy2vqx...  \n",
       "4  http://mmsns.qpic.cn/mmsns/A0R6pEEjricaFJgpW6w...  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<div class=\"weui-desktop-vm_primary\"><img src=\"http://mmbiz.qpic.cn/mmbiz_png/cdSLicavGUr23RakfD1j8csT3LMhYpOQ21ECXeZMpic1k1dTBWprafgazS97J5JaVCHNQr4ib0cuSGVDWc9rJUiatQ/0?wx_fmt=png\" class=\"inner_link_account_avatar\"> <strong class=\"inner_link_account_nickname\">腾讯媒体研究院</strong> <i class=\"inner_link_account_wechat\">微信号：TencentMRI</i></div> <div class=\"weui-desktop-vm_default inner_link_account_type\">订阅号</div>\n"
     ]
    }
   ],
   "source": [
    "element = driver.find_element_by_xpath('//ul[@class=\"inner_link_account_list\"]/li')\n",
    "main_content = element.get_attribute('innerHTML')\n",
    "print(main_content)\n",
    "element.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n跳转_input = driver.find_element_by_xpath(\\'//span[@class=\"weui-desktop-pagination__form\"]/input\\')\\n跳转_a = driver.find_element_by_xpath(\\'//span[@class=\"weui-desktop-pagination__form\"]/a\\')\\n跳转_input.clear()\\n跳转_input.send_keys(2)\\n跳转_a.click()\\n'"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 跳转testing\n",
    "'''\n",
    "跳转_input = driver.find_element_by_xpath('//span[@class=\"weui-desktop-pagination__form\"]/input')\n",
    "跳转_a = driver.find_element_by_xpath('//span[@class=\"weui-desktop-pagination__form\"]/a')\n",
    "跳转_input.clear()\n",
    "跳转_input.send_keys(2)\n",
    "跳转_a.click()\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 117]\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "# 跳转上限\n",
    "l_e = driver.find_elements_by_xpath('//label[@class=\"weui-desktop-pagination__num\"]')\n",
    "l_e_int  = [int(x.text) for x in l_e] \n",
    "print (l_e_int)\n",
    "print (l_e_int[0]==l_e_int[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117]\n"
     ]
    }
   ],
   "source": [
    "pages = list(range(l_e_int[0],l_e_int[-1]+1 ))\n",
    "#print(pages[0:2])\n",
    "pages = list(range(1,l_e_int[-1]+1 ))\n",
    "print(pages)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 循环/遍历"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# global varialbes \n",
    "html_raw = dict()\n",
    "main_content =\"\"\n",
    "element = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_pages (pages):\n",
    "    for p in pages:\n",
    "        print (p,end='\\t')\n",
    "\n",
    "        跳转_input = driver.find_element_by_xpath('//span[@class=\"weui-desktop-pagination__form\"]/input')\n",
    "        跳转_a = driver.find_element_by_xpath('//span[@class=\"weui-desktop-pagination__form\"]/a')\n",
    "        跳转_input.clear()\n",
    "        跳转_input.send_keys(p)\n",
    "        跳转_a.click()\n",
    "\n",
    "        time.sleep(5+10*random())\n",
    "\n",
    "        element = driver.find_element_by_xpath('//div[@class=\"inner_link_article_list\"]')\n",
    "        main_content = element.get_attribute('innerHTML')\n",
    "        #print(main_content)\n",
    "        html_raw[p] = main_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\t2\t3\t4\t5\t6\t7\t8\t9\t10\t11\t12\t13\t14\t15\t16\t17\t18\t19\t20\t21\t22\t23\t24\t25\t26\t27\t28\t29\t30\t31\t32\t33\t34\t35\t36\t37\t38\t39\t40\t41\t42\t43\t44\t45\t46\t47\t48\t49\t50\t51\t52\t53\t54\t55\t56\t57\t58\t59\t60\t61\t62\t63\t64\t65\t66\t67\t68\t69\t70\t71\t72\t73\t74\t75\t76\t77\t78\t79\t80\t81\t82\t83\t84\t85\t86\t87\t88\t89\t90\t91\t92\t93\t94\t95\t96\t97\t98\t99\t100\t101\t102\t103\t104\t105\t106\t107\t108\t109\t110\t111\t112\t113\t114\t115\t116\t117\t"
     ]
    }
   ],
   "source": [
    "process_pages(pages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>html_snippets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>115</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>117 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         html_snippets\n",
       "1    <div><label class=\"inner_link_article_item\"><s...\n",
       "2    <div><label class=\"inner_link_article_item\"><s...\n",
       "3    <div><label class=\"inner_link_article_item\"><s...\n",
       "4    <div><label class=\"inner_link_article_item\"><s...\n",
       "5    <div><label class=\"inner_link_article_item\"><s...\n",
       "..                                                 ...\n",
       "113  <div><label class=\"inner_link_article_item\"><s...\n",
       "114  <div><label class=\"inner_link_article_item\"><s...\n",
       "115  <div><label class=\"inner_link_article_item\"><s...\n",
       "116  <div><label class=\"inner_link_article_item\"><s...\n",
       "117  <div><label class=\"inner_link_article_item\"><s...\n",
       "\n",
       "[117 rows x 1 columns]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame([html_raw]).T\n",
    "df.columns = [\"html_snippets\"]\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'html_raw' (dict)\n"
     ]
    }
   ],
   "source": [
    "%store html_raw\n",
    "import pickle \n",
    "filehandler = open(\"html_raw\", 'wb') \n",
    "pickle.dump(html_raw, filehandler)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "79\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>html_snippets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>104</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>107</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>109</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>111</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>115</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>&lt;div&gt;&lt;label class=\"inner_link_article_item\"&gt;&lt;s...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         html_snippets\n",
       "12   <div><label class=\"inner_link_article_item\"><s...\n",
       "81   <div><label class=\"inner_link_article_item\"><s...\n",
       "82   <div><label class=\"inner_link_article_item\"><s...\n",
       "83   <div><label class=\"inner_link_article_item\"><s...\n",
       "84   <div><label class=\"inner_link_article_item\"><s...\n",
       "85   <div><label class=\"inner_link_article_item\"><s...\n",
       "86   <div><label class=\"inner_link_article_item\"><s...\n",
       "87   <div><label class=\"inner_link_article_item\"><s...\n",
       "88   <div><label class=\"inner_link_article_item\"><s...\n",
       "89   <div><label class=\"inner_link_article_item\"><s...\n",
       "90   <div><label class=\"inner_link_article_item\"><s...\n",
       "91   <div><label class=\"inner_link_article_item\"><s...\n",
       "92   <div><label class=\"inner_link_article_item\"><s...\n",
       "93   <div><label class=\"inner_link_article_item\"><s...\n",
       "94   <div><label class=\"inner_link_article_item\"><s...\n",
       "95   <div><label class=\"inner_link_article_item\"><s...\n",
       "96   <div><label class=\"inner_link_article_item\"><s...\n",
       "97   <div><label class=\"inner_link_article_item\"><s...\n",
       "98   <div><label class=\"inner_link_article_item\"><s...\n",
       "99   <div><label class=\"inner_link_article_item\"><s...\n",
       "100  <div><label class=\"inner_link_article_item\"><s...\n",
       "101  <div><label class=\"inner_link_article_item\"><s...\n",
       "102  <div><label class=\"inner_link_article_item\"><s...\n",
       "103  <div><label class=\"inner_link_article_item\"><s...\n",
       "104  <div><label class=\"inner_link_article_item\"><s...\n",
       "105  <div><label class=\"inner_link_article_item\"><s...\n",
       "106  <div><label class=\"inner_link_article_item\"><s...\n",
       "107  <div><label class=\"inner_link_article_item\"><s...\n",
       "108  <div><label class=\"inner_link_article_item\"><s...\n",
       "109  <div><label class=\"inner_link_article_item\"><s...\n",
       "110  <div><label class=\"inner_link_article_item\"><s...\n",
       "111  <div><label class=\"inner_link_article_item\"><s...\n",
       "112  <div><label class=\"inner_link_article_item\"><s...\n",
       "113  <div><label class=\"inner_link_article_item\"><s...\n",
       "114  <div><label class=\"inner_link_article_item\"><s...\n",
       "115  <div><label class=\"inner_link_article_item\"><s...\n",
       "116  <div><label class=\"inner_link_article_item\"><s...\n",
       "117  <div><label class=\"inner_link_article_item\"><s..."
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_out = df[~df.duplicated()]\n",
    "print (len(df_out))\n",
    "df[df.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[12,\n",
       " 81,\n",
       " 82,\n",
       " 83,\n",
       " 84,\n",
       " 85,\n",
       " 86,\n",
       " 87,\n",
       " 88,\n",
       " 89,\n",
       " 90,\n",
       " 91,\n",
       " 92,\n",
       " 93,\n",
       " 94,\n",
       " 95,\n",
       " 96,\n",
       " 97,\n",
       " 98,\n",
       " 99,\n",
       " 100,\n",
       " 101,\n",
       " 102,\n",
       " 103,\n",
       " 104,\n",
       " 105,\n",
       " 106,\n",
       " 107,\n",
       " 108,\n",
       " 109,\n",
       " 110,\n",
       " 111,\n",
       " 112,\n",
       " 113,\n",
       " 114,\n",
       " 115,\n",
       " 116,\n",
       " 117]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "try_again = list(df[df.duplicated()].index)\n",
    "print(try_again)\n",
    "try_again = try_again + list (set(pages).difference(set(df.index.values)))\n",
    "try_again"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 暂存档"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = fn [\"output\"] [\"公众号_htm_snippets\"] \n",
    "df_out.to_csv(filename.format(公众号=公众号), sep=\"\\t\", encoding=\"utf8\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6,7,8,5,6,6,5,5,5,5,6,7,6,5,9,7,6,6,5,6,6,5,6,5,5,6,6,6,7,5,5,5,6,6,5,5,5,5,5,6,6,6,8,5,6,6,6,5,6,6,7,6,7,11,5,5,5,6,9,6,5,5,5,10,5,6,6,9,6,5,5,10,11,5,5,5,5,6,5,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,"
     ]
    }
   ],
   "source": [
    "def parse_html_snippets(_snippet_):\n",
    "    root = fromstring(_snippet_) \n",
    "    title = [x.text for x in root.xpath('//div[@class=\"inner_link_article_title\"]')]\n",
    "    create_time = [x.text for x in root.xpath('//div[@class=\"inner_link_article_date\"]')]\n",
    "    link = [x for x in root.xpath('//a/@href')]\n",
    "    _df_ = pd.DataFrame({\"title\":title, \"create_time\": create_time, \"link\":link})\n",
    "    return(_df_)\n",
    "    \n",
    "l_df = []\n",
    "for p in pages:\n",
    "    _df_ = parse_html_snippets(df.loc[p,\"html_snippets\"])\n",
    "    print (len(_df_), end=\",\")\n",
    "    l_df.append(_df_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>媒体融合语境下的短视频丨【芒种·观点】</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>腾讯新闻热点运营主编邱帅：爆款资讯短视频的运营逻辑丨【芒种·观点】</td>\n",
       "      <td>2020-05-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>腾讯深网主编范晓东：科技大佬访谈的运营之道丨课程预告</td>\n",
       "      <td>2020-05-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>如何频繁产出爆款新闻短视频？丨【芒种·观点】</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>你所追寻的“价值”是什么？</td>\n",
       "      <td>2020-05-12</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>700</th>\n",
       "      <td>人工智能会将新闻媒体带向何方？看看纽约时报、路透社和其他媒体巨头的人工智能应用丨【芒种·趋势】</td>\n",
       "      <td>2018-08-31</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>701</th>\n",
       "      <td>176位专家对新闻业的预测：媒体未来的8大趋势丨【芒种·趋势】</td>\n",
       "      <td>2018-08-29</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>702</th>\n",
       "      <td>三四线用户互联网生活报告：娱乐消遣、追随从众丨【芒种·报告】</td>\n",
       "      <td>2018-08-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>703</th>\n",
       "      <td>方军：我们可能低估了知识产品的媒体属性丨【芒种·观点】</td>\n",
       "      <td>2018-08-27</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>704</th>\n",
       "      <td>今天，首都媒体界有一件大事！北京广播电视台融媒体中心成立啦！</td>\n",
       "      <td>2018-08-27</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>705 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               title create_time  \\\n",
       "0                                媒体融合语境下的短视频丨【芒种·观点】  2020-05-15   \n",
       "1                  腾讯新闻热点运营主编邱帅：爆款资讯短视频的运营逻辑丨【芒种·观点】  2020-05-14   \n",
       "2                         腾讯深网主编范晓东：科技大佬访谈的运营之道丨课程预告  2020-05-14   \n",
       "3                             如何频繁产出爆款新闻短视频？丨【芒种·观点】  2020-05-13   \n",
       "4                                      你所追寻的“价值”是什么？  2020-05-12   \n",
       "..                                               ...         ...   \n",
       "700  人工智能会将新闻媒体带向何方？看看纽约时报、路透社和其他媒体巨头的人工智能应用丨【芒种·趋势】  2018-08-31   \n",
       "701                  176位专家对新闻业的预测：媒体未来的8大趋势丨【芒种·趋势】  2018-08-29   \n",
       "702                   三四线用户互联网生活报告：娱乐消遣、追随从众丨【芒种·报告】  2018-08-28   \n",
       "703                      方军：我们可能低估了知识产品的媒体属性丨【芒种·观点】  2018-08-27   \n",
       "704                   今天，首都媒体界有一件大事！北京广播电视台融媒体中心成立啦！  2018-08-27   \n",
       "\n",
       "                                                  link  \n",
       "0    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "1    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "2    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "3    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "4    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "..                                                 ...  \n",
       "700  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "701  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "702  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "703  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "704  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "\n",
       "[705 rows x 3 columns]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 抓取全部文章的标题、时间、链接\n",
    "df_url_out = pd.concat(l_df).reset_index(drop=True)\n",
    "df_url_out.loc[0:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>value</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>你所追寻的“价值”是什么？</td>\n",
       "      <td>2020-05-12</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>粉丝节现场先睹为快！与《中国人的一天》一起回顾十年故事丨鹅星人粉丝节</td>\n",
       "      <td>2020-01-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>《刺死辱母者》记者王瑞锋：采访是从人心到人心的过程，也是一场抵达丨【芒种·人物】</td>\n",
       "      <td>2020-01-10</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>听重庆棒棒父子的幕后故事，《中国人的一天》粉丝见面会入选名单公布 丨 鹅星人粉丝节</td>\n",
       "      <td>2020-01-09</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>“一句话”涨100万粉：拥粉2500万“贫穷料理”崛起背后丨【芒种·案例】</td>\n",
       "      <td>2020-01-09</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>696</th>\n",
       "      <td>三四线用户互联网生活报告：娱乐消遣、追随从众丨【芒种·报告】</td>\n",
       "      <td>2018-08-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>698</th>\n",
       "      <td>今天，首都媒体界有一件大事！北京广播电视台融媒体中心成立啦！</td>\n",
       "      <td>2018-08-27</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>699</th>\n",
       "      <td>美联社报告：数字时代媒体创收新途径 丨【芒种·报告】</td>\n",
       "      <td>2018-09-03</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>702</th>\n",
       "      <td>三四线用户互联网生活报告：娱乐消遣、追随从众丨【芒种·报告】</td>\n",
       "      <td>2018-08-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>704</th>\n",
       "      <td>今天，首都媒体界有一件大事！北京广播电视台融媒体中心成立啦！</td>\n",
       "      <td>2018-08-27</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>357 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           title create_time  \\\n",
       "value                                                          \n",
       "4                                  你所追寻的“价值”是什么？  2020-05-12   \n",
       "8             粉丝节现场先睹为快！与《中国人的一天》一起回顾十年故事丨鹅星人粉丝节  2020-01-14   \n",
       "10      《刺死辱母者》记者王瑞锋：采访是从人心到人心的过程，也是一场抵达丨【芒种·人物】  2020-01-10   \n",
       "11     听重庆棒棒父子的幕后故事，《中国人的一天》粉丝见面会入选名单公布 丨 鹅星人粉丝节  2020-01-09   \n",
       "12         “一句话”涨100万粉：拥粉2500万“贫穷料理”崛起背后丨【芒种·案例】  2020-01-09   \n",
       "...                                          ...         ...   \n",
       "696               三四线用户互联网生活报告：娱乐消遣、追随从众丨【芒种·报告】  2018-08-28   \n",
       "698               今天，首都媒体界有一件大事！北京广播电视台融媒体中心成立啦！  2018-08-27   \n",
       "699                   美联社报告：数字时代媒体创收新途径 丨【芒种·报告】  2018-09-03   \n",
       "702               三四线用户互联网生活报告：娱乐消遣、追随从众丨【芒种·报告】  2018-08-28   \n",
       "704               今天，首都媒体界有一件大事！北京广播电视台融媒体中心成立啦！  2018-08-27   \n",
       "\n",
       "                                                    link  \n",
       "value                                                     \n",
       "4      http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "8      http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "10     http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "11     http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "12     http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "...                                                  ...  \n",
       "696    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "698    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "699    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "702    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "704    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "\n",
       "[357 rows x 3 columns]"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# tagging 标记\n",
    "tagging_list = [\"\",\"马化腾\", \"腾讯\", \"微众银行\",\"腾讯复星\",\"腾讯风控\",\"腾讯支付\",\"WeChat\",\"We Remit\",\\\n",
    "                \"新媒体\",\"自媒体\",\"产品经理\",\"字节跳动\"\\\n",
    "                \"腾讯财付通\",\"鹅厂\",\"QQ钱包\",\"QQ红包\",\"QQ\",\\\n",
    "                \"移动直播\",\"大咖\",\"短视频\",\"网红经济\",\"直播带货\",\"文娱产业\",\\\n",
    "                \"爆款\", \"产品\",\"用户需求文档\",\"人工智能\",\"算法\",\\\n",
    "                \"泛娱乐\",\"艾瑞\",\"知识付费\",\"大数据\",\"云\",\\\n",
    "                \"区块链\",\"金融云\",\"O2O\",\"农产品\",\"家乡\",\\\n",
    "                \"数据\", \"数据赋能\", \"智能\", \"数字孪生\", \"智慧大脑\",\\\n",
    "                \"微信\", \"微信支付\", \"跨境支付\", \"移动支付\",\"非银行支付\",\"电子支付\",\\\n",
    "                \"互联网金融\", \"金融科技\",\"互联网＋\",\"互联网+金融\",\"普惠金融\",\"虚拟银行\",\\\n",
    "                \"科技向善\",\"开放\",\"生态\",\\\n",
    "                \"城市\", \"粤港澳大湾区\", \"平台\", \"可持续发展\", \"未来\", \"绿色\",\\\n",
    "                \"疫情\", \"复工\",\"停课\",\"宅经济\",\\\n",
    "                \"传媒\"]\n",
    "\n",
    "v_v_list = []\n",
    "\n",
    "for tag in tagging_list:\n",
    "    index_list = df_url_out [ df_url_out.title.str.contains(tag) ].index.tolist()\n",
    "    v_v_pairs = pd.DataFrame({tag:index_list}).melt().set_index(\"value\")\n",
    "    v_v_list.append(v_v_pairs)\n",
    "\n",
    "df_cat = v_v_list[0]\n",
    "for d in v_v_list:\n",
    "    df_cat.update(d)\n",
    "    \n",
    "# 尚未标记内容\n",
    "df_url_out.loc [ df_cat.query('variable==\"\"').index ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>未来新闻业变革方向：改变运作模式，追求技术共识丨【芒种·趋势】</td>\n",
       "      <td>2020-01-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>腾讯证券12月优秀合作伙伴揭晓，热文TOP10新鲜出炉丨内容势力榜</td>\n",
       "      <td>2020-01-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>粉丝节现场先睹为快！与《中国人的一天》一起回顾十年故事丨鹅星人粉丝节</td>\n",
       "      <td>2020-01-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>一晚70亿阅读量，单篇涨粉17万，GQ打造爆款内容的幕后故事丨【芒种·观点】</td>\n",
       "      <td>2020-01-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>《刺死辱母者》记者王瑞锋：采访是从人心到人心的过程，也是一场抵达丨【芒种·人物】</td>\n",
       "      <td>2020-01-10</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>700</th>\n",
       "      <td>人工智能会将新闻媒体带向何方？看看纽约时报、路透社和其他媒体巨头的人工智能应用丨【芒种·趋势】</td>\n",
       "      <td>2018-08-31</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>701</th>\n",
       "      <td>176位专家对新闻业的预测：媒体未来的8大趋势丨【芒种·趋势】</td>\n",
       "      <td>2018-08-29</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>702</th>\n",
       "      <td>三四线用户互联网生活报告：娱乐消遣、追随从众丨【芒种·报告】</td>\n",
       "      <td>2018-08-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>703</th>\n",
       "      <td>方军：我们可能低估了知识产品的媒体属性丨【芒种·观点】</td>\n",
       "      <td>2018-08-27</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>704</th>\n",
       "      <td>今天，首都媒体界有一件大事！北京广播电视台融媒体中心成立啦！</td>\n",
       "      <td>2018-08-27</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>229 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               title create_time  \\\n",
       "64                   未来新闻业变革方向：改变运作模式，追求技术共识丨【芒种·趋势】  2020-01-15   \n",
       "65                 腾讯证券12月优秀合作伙伴揭晓，热文TOP10新鲜出炉丨内容势力榜  2020-01-14   \n",
       "66                粉丝节现场先睹为快！与《中国人的一天》一起回顾十年故事丨鹅星人粉丝节  2020-01-14   \n",
       "67            一晚70亿阅读量，单篇涨粉17万，GQ打造爆款内容的幕后故事丨【芒种·观点】  2020-01-13   \n",
       "68          《刺死辱母者》记者王瑞锋：采访是从人心到人心的过程，也是一场抵达丨【芒种·人物】  2020-01-10   \n",
       "..                                               ...         ...   \n",
       "700  人工智能会将新闻媒体带向何方？看看纽约时报、路透社和其他媒体巨头的人工智能应用丨【芒种·趋势】  2018-08-31   \n",
       "701                  176位专家对新闻业的预测：媒体未来的8大趋势丨【芒种·趋势】  2018-08-29   \n",
       "702                   三四线用户互联网生活报告：娱乐消遣、追随从众丨【芒种·报告】  2018-08-28   \n",
       "703                      方军：我们可能低估了知识产品的媒体属性丨【芒种·观点】  2018-08-27   \n",
       "704                   今天，首都媒体界有一件大事！北京广播电视台融媒体中心成立啦！  2018-08-27   \n",
       "\n",
       "                                                  link  \n",
       "64   http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "65   http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "66   http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "67   http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "68   http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "..                                                 ...  \n",
       "700  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "701  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "702  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "703  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "704  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "\n",
       "[229 rows x 3 columns]"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_url_out[df_url_out.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>媒体融合语境下的短视频丨【芒种·观点】</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>腾讯新闻热点运营主编邱帅：爆款资讯短视频的运营逻辑丨【芒种·观点】</td>\n",
       "      <td>2020-05-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>腾讯深网主编范晓东：科技大佬访谈的运营之道丨课程预告</td>\n",
       "      <td>2020-05-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>如何频繁产出爆款新闻短视频？丨【芒种·观点】</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>你所追寻的“价值”是什么？</td>\n",
       "      <td>2020-05-12</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>478</th>\n",
       "      <td>人工智能会将新闻媒体带向何方？看看纽约时报、路透社和其他媒体巨头的人工智能应用丨【芒种·趋势】</td>\n",
       "      <td>2018-08-31</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>479</th>\n",
       "      <td>176位专家对新闻业的预测：媒体未来的8大趋势丨【芒种·趋势】</td>\n",
       "      <td>2018-08-29</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>480</th>\n",
       "      <td>三四线用户互联网生活报告：娱乐消遣、追随从众丨【芒种·报告】</td>\n",
       "      <td>2018-08-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>481</th>\n",
       "      <td>方军：我们可能低估了知识产品的媒体属性丨【芒种·观点】</td>\n",
       "      <td>2018-08-27</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>482</th>\n",
       "      <td>今天，首都媒体界有一件大事！北京广播电视台融媒体中心成立啦！</td>\n",
       "      <td>2018-08-27</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>476 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               title create_time  \\\n",
       "0                                媒体融合语境下的短视频丨【芒种·观点】  2020-05-15   \n",
       "1                  腾讯新闻热点运营主编邱帅：爆款资讯短视频的运营逻辑丨【芒种·观点】  2020-05-14   \n",
       "2                         腾讯深网主编范晓东：科技大佬访谈的运营之道丨课程预告  2020-05-14   \n",
       "3                             如何频繁产出爆款新闻短视频？丨【芒种·观点】  2020-05-13   \n",
       "4                                      你所追寻的“价值”是什么？  2020-05-12   \n",
       "..                                               ...         ...   \n",
       "478  人工智能会将新闻媒体带向何方？看看纽约时报、路透社和其他媒体巨头的人工智能应用丨【芒种·趋势】  2018-08-31   \n",
       "479                  176位专家对新闻业的预测：媒体未来的8大趋势丨【芒种·趋势】  2018-08-29   \n",
       "480                   三四线用户互联网生活报告：娱乐消遣、追随从众丨【芒种·报告】  2018-08-28   \n",
       "481                      方军：我们可能低估了知识产品的媒体属性丨【芒种·观点】  2018-08-27   \n",
       "482                   今天，首都媒体界有一件大事！北京广播电视台融媒体中心成立啦！  2018-08-27   \n",
       "\n",
       "                                                  link  \n",
       "0    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "1    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "2    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "3    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "4    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "..                                                 ...  \n",
       "478  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "479  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "480  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "481  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "482  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...  \n",
       "\n",
       "[476 rows x 3 columns]"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_url_out[~df_url_out.duplicated()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "      <th>variable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>媒体融合语境下的短视频丨【芒种·观点】</td>\n",
       "      <td>2020-05-15</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>短视频</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>腾讯新闻热点运营主编邱帅：爆款资讯短视频的运营逻辑丨【芒种·观点】</td>\n",
       "      <td>2020-05-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>爆款</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>腾讯深网主编范晓东：科技大佬访谈的运营之道丨课程预告</td>\n",
       "      <td>2020-05-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>腾讯</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>如何频繁产出爆款新闻短视频？丨【芒种·观点】</td>\n",
       "      <td>2020-05-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>爆款</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>你所追寻的“价值”是什么？</td>\n",
       "      <td>2020-05-12</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>700</th>\n",
       "      <td>人工智能会将新闻媒体带向何方？看看纽约时报、路透社和其他媒体巨头的人工智能应用丨【芒种·趋势】</td>\n",
       "      <td>2018-08-31</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>智能</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>701</th>\n",
       "      <td>176位专家对新闻业的预测：媒体未来的8大趋势丨【芒种·趋势】</td>\n",
       "      <td>2018-08-29</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>未来</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>702</th>\n",
       "      <td>三四线用户互联网生活报告：娱乐消遣、追随从众丨【芒种·报告】</td>\n",
       "      <td>2018-08-28</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>703</th>\n",
       "      <td>方军：我们可能低估了知识产品的媒体属性丨【芒种·观点】</td>\n",
       "      <td>2018-08-27</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>产品</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>704</th>\n",
       "      <td>今天，首都媒体界有一件大事！北京广播电视台融媒体中心成立啦！</td>\n",
       "      <td>2018-08-27</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>无法分类</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>705 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               title create_time  \\\n",
       "0                                媒体融合语境下的短视频丨【芒种·观点】  2020-05-15   \n",
       "1                  腾讯新闻热点运营主编邱帅：爆款资讯短视频的运营逻辑丨【芒种·观点】  2020-05-14   \n",
       "2                         腾讯深网主编范晓东：科技大佬访谈的运营之道丨课程预告  2020-05-14   \n",
       "3                             如何频繁产出爆款新闻短视频？丨【芒种·观点】  2020-05-13   \n",
       "4                                      你所追寻的“价值”是什么？  2020-05-12   \n",
       "..                                               ...         ...   \n",
       "700  人工智能会将新闻媒体带向何方？看看纽约时报、路透社和其他媒体巨头的人工智能应用丨【芒种·趋势】  2018-08-31   \n",
       "701                  176位专家对新闻业的预测：媒体未来的8大趋势丨【芒种·趋势】  2018-08-29   \n",
       "702                   三四线用户互联网生活报告：娱乐消遣、追随从众丨【芒种·报告】  2018-08-28   \n",
       "703                      方军：我们可能低估了知识产品的媒体属性丨【芒种·观点】  2018-08-27   \n",
       "704                   今天，首都媒体界有一件大事！北京广播电视台融媒体中心成立啦！  2018-08-27   \n",
       "\n",
       "                                                  link variable  \n",
       "0    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...      短视频  \n",
       "1    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...       爆款  \n",
       "2    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...       腾讯  \n",
       "3    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...       爆款  \n",
       "4    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...     无法分类  \n",
       "..                                                 ...      ...  \n",
       "700  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...       智能  \n",
       "701  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...       未来  \n",
       "702  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...     无法分类  \n",
       "703  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...       产品  \n",
       "704  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...     无法分类  \n",
       "\n",
       "[705 rows x 4 columns]"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_o = df_url_out.join(df_cat).replace(\"\", np.nan).fillna(\"无法分类\")\n",
    "df_o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>create_time</th>\n",
       "      <th>link</th>\n",
       "      <th>variable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>腾讯新闻热点运营主编邱帅：爆款资讯短视频的运营逻辑丨【芒种·观点】</td>\n",
       "      <td>2020-05-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>爆款</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>腾讯深网主编范晓东：科技大佬访谈的运营之道丨课程预告</td>\n",
       "      <td>2020-05-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>腾讯</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>腾讯证券12月优秀合作伙伴揭晓，热文TOP10新鲜出炉丨内容势力榜</td>\n",
       "      <td>2020-01-14</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>腾讯</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>揭秘腾讯新闻内容生产方法论，5位大咖导师带着真知灼见等你来撩丨课程免费报名</td>\n",
       "      <td>2020-04-27</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>大咖</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>解析腾讯新闻内容运营逻辑：从算法到偏好，怎样的内容有流量？丨【芒种·案例】</td>\n",
       "      <td>2020-04-20</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>算法</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>445</th>\n",
       "      <td>腾讯视频产品负责人分享弯道超车终极秘诀丨【芒种·案例】</td>\n",
       "      <td>2018-10-22</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>产品</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>451</th>\n",
       "      <td>“干货”满满！腾讯新闻短视频训练营圆满落幕</td>\n",
       "      <td>2018-10-13</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>短视频</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>456</th>\n",
       "      <td>腾讯新闻8月短视频机构榜单强势来袭，短视频专项训练营即将开启！</td>\n",
       "      <td>2018-10-08</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>短视频</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>461</th>\n",
       "      <td>2018移动视频十大观察丨腾讯方法论【芒种·趋势】</td>\n",
       "      <td>2018-09-26</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>腾讯</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>471</th>\n",
       "      <td>腾讯网总编辑李方：推进主旋律生产力，助力媒体融合发展丨【芒种·观点】</td>\n",
       "      <td>2018-09-11</td>\n",
       "      <td>http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...</td>\n",
       "      <td>腾讯</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>105 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     title create_time  \\\n",
       "1        腾讯新闻热点运营主编邱帅：爆款资讯短视频的运营逻辑丨【芒种·观点】  2020-05-14   \n",
       "2               腾讯深网主编范晓东：科技大佬访谈的运营之道丨课程预告  2020-05-14   \n",
       "7        腾讯证券12月优秀合作伙伴揭晓，热文TOP10新鲜出炉丨内容势力榜  2020-01-14   \n",
       "13   揭秘腾讯新闻内容生产方法论，5位大咖导师带着真知灼见等你来撩丨课程免费报名  2020-04-27   \n",
       "20   解析腾讯新闻内容运营逻辑：从算法到偏好，怎样的内容有流量？丨【芒种·案例】  2020-04-20   \n",
       "..                                     ...         ...   \n",
       "445            腾讯视频产品负责人分享弯道超车终极秘诀丨【芒种·案例】  2018-10-22   \n",
       "451                  “干货”满满！腾讯新闻短视频训练营圆满落幕  2018-10-13   \n",
       "456        腾讯新闻8月短视频机构榜单强势来袭，短视频专项训练营即将开启！  2018-10-08   \n",
       "461              2018移动视频十大观察丨腾讯方法论【芒种·趋势】  2018-09-26   \n",
       "471     腾讯网总编辑李方：推进主旋律生产力，助力媒体融合发展丨【芒种·观点】  2018-09-11   \n",
       "\n",
       "                                                  link variable  \n",
       "1    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...       爆款  \n",
       "2    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...       腾讯  \n",
       "7    http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...       腾讯  \n",
       "13   http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...       大咖  \n",
       "20   http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...       算法  \n",
       "..                                                 ...      ...  \n",
       "445  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...       产品  \n",
       "451  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...      短视频  \n",
       "456  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...      短视频  \n",
       "461  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...       腾讯  \n",
       "471  http://mp.weixin.qq.com/s?__biz=MzIzNzYwMzM3Ng...       腾讯  \n",
       "\n",
       "[105 rows x 4 columns]"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_o[df_o.title.str.contains(\"腾讯\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>variable</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>无法分类</th>\n",
       "      <td>357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>腾讯</th>\n",
       "      <td>60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>未来</th>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>智能</th>\n",
       "      <td>46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>产品</th>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>短视频</th>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>数据</th>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>传媒</th>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>爆款</th>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>平台</th>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>生态</th>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>新媒体</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>知识付费</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>算法</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>大咖</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>云</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>城市</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>疫情</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>直播带货</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>微信</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>开放</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>泛娱乐</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          title\n",
       "variable       \n",
       "无法分类        357\n",
       "腾讯           60\n",
       "未来           50\n",
       "智能           46\n",
       "产品           44\n",
       "短视频          40\n",
       "数据           26\n",
       "传媒           15\n",
       "爆款           14\n",
       "平台           13\n",
       "生态            8\n",
       "新媒体           7\n",
       "知识付费          7\n",
       "算法            5\n",
       "大咖            3\n",
       "云             3\n",
       "城市            2\n",
       "疫情            1\n",
       "直播带货          1\n",
       "微信            1\n",
       "开放            1\n",
       "泛娱乐           1"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_stats = df_o.groupby(by=\"variable\").agg({\"title\":\"count\"}).sort_values(by=\"title\", ascending=False)\n",
    "df_stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_account.columns.name = \"rel_accounts\"\n",
    "df_o.columns.name = \"url_cat\"\n",
    "df_stats.columns.name = \"stats\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "_df_.columns.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_o.to_excel(fn[\"output\"][\"公众号_数据_xlsx\"].format(公众号=公众号))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
